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ABSTRACT 

In the 1992~93 school year, in response to a request 
from the Board of Trustees of the Austin Independent School District 
(Texas) (AISD) , the district's Office of Research and Evaluation 
conducted a retrospective examination of the 1991-92 AISD programs to 
determine cost-effectiveness when possible. This report describes the 
review process. Cost-effectiveness was calculated by dividing a 
measure of cost in dollars by one of two measures of effect: 
achievement or not dropping out. Cost was defined as the program's 
appropriation (budget). Acnievement was based on scores on the 
Norm-Referenced Assessment Program for Texas or the Iowa Tests of 
Basic Skills. Dropout information came from the district's system or 
comparisons of predicted and actual dropouts. The office reviewed 85 
programs or program components, with cost-effectiveness calculated 
for 18 by achievement, 16 by drorvout measures, and the balance rated 
on other measures of effectiveness. Most programs were rated as 
effective. Many programs with high initial costs were relatively low 
in cost per. student once established. The cost-effectiveness of 
volunteer services was also demonstrated* Twenty-seven tables present 
study data. (Contains 12 references.) (SLD) 
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ABSTRACT 



Over a number of years, the Offee of Research and Evaluation (ORE) has provided the Board of 
Trustees of the Austin Independent School District (AISD) with comparisons of the effectiveness of 
many of the District's special programs. In the 1 992-93 school year, at the Board's request to provide 
it with a measure of effect as weEl as cost in the program effectiveness charts ORE prepares for the 
Board's annua! budget study session, ORE conducted a retrospective examination of 1991-92 AISD 
programs. In February 1993, ORE presented the Board with program effectiveness charts which 
included cost-effectiveness ratios for many programs evaluated during 1991-92. Comments and 
suggestions from the Board of Trustees, District staff, and community members were used to finetune 
the methodology and the format in which the information was reported. A final report was issued in 
May 1993. 

Cost-effectiveness was calculated by dividing a measure of cost in dollars by one of two measures of 
effect: (1 ) achievement, or (2) not dropping out. The cost of a program was defined as the program's 
appropriation (i.e., budget). The achievement measure of effect was based on standardized test scores 
from either the Norm-referenced Assessment Program for Texas (NAPT) or the Iowa Tests of basic 
Skills (ITBS). The dropout prevention measure of effect was derived from the comparison in ORE's 
generic evaluation system (GENESYS) of the number rf students in a program predicted to drop out 
with the actual number who did drop out. The difference between the predicted and actual number 
of dropouts, the number of students kept in school, is the dropout prevention effect measure. The 
cost-effectiveness ratio, expressed in dollars, which results from dividing cost by effect (C/E) is a 
measure of the cost-effectiveness of a program, i.e., the amount of effect for monies expended. 

Where cost or effect measures were not obtainable, and other evaluation information about a program 
was available, ORE staff assigned ratings of effectiveness to the programs evaluated based on other 
indicators, such as survey results, retention rate, and attendance rate. 

The major findings of the study were: 

1. ORE reviewed 85 1991-92 programs or program components. Cost-effectiveness was 
calculated for 18 programs using an achievement effect measure and for 16 programs using 
a dropout prevention effect measure. An additional 14 dropout prevention programs were rated 
on effectiveness, although cost information could not be obtained. Another 37 programs were 
rated on effectiveness based on other evaluation information. 

2. Most programs evaluated in 1 991-92 in AISD were rated as effective. Approximately 21 % of 
the ratings were based on achievement, 35% were based on the number of students not 
dropping out, and 44% were based on- other evaluation findings. 

3. In general, the programs showing the highest achievement gains for students tend to be 
programs that offer students enriching experiences in addition to the regular curriculum. Most 
of these programs have a relatively high initial cost, but once the program is in place the cost 
for the per-pupil gain is relatively low. 

4. A common feature among successful dropout prevention pt ^grams is that they provide students 
with individual attention or the possibility of flexibility in class schedules and enrichment 
activities. Many of these programs are dependent on the use of volunteers or mentors, so they 
would not be as cost-effective if the District were to purchase the same services. 
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RIDING THEM OFF INTO THE SUNSET: 
A COST-EFFECTSVENESS REVIEW OF A DISTRICT'S SPECIAL PROGRAMS 



INTRODUCTION 

What works in public education, and can we afford it? These questions are being asked more 
frequently in Austin and around the country as accountability is emphasized during a time of tightening 
budgets. In August 1 992, the Board of Trustees of the Austin, Texas public schools (AISD) challenged 
the Office of Research and Evaluation (ORE) to provide it with measures of both effect and cost for 
each of the special programs evaluated by ORE. Implicit in the Board's request, although ORE was not 
charged with developing a process, was the concept of sunset review, through which ineffective or 
overly costly programs could be identified and funding withdrawn. 

This request jibed with the thinking 
and efforts of the evaluation staff, 
who had over a number of years 
incorporated cost information into 
evaluation reports to accompany 
the effectiveness information 
already being reported. To move 
beyond current reporting toward 
true cost-effectiveness analysis 
and to lay the groundwork for a 
sunset review process for all 
programs was, however, an 
enormous challenge --both 
methodologically and, regrettably, 
politically. 

In principle, calculating cost- 
effectiveness looked to be a 
straightforward procedure. ORE 
was already accustomed to reporting cost information, as well as considerable effectiveness 
information, but the difficulty lay in combining the measures. Although ample literature about cost 
analysis existed for other disciplines, very few texts were devoted to applications in an education 
setting, and these contained onfy general guidelines and very simple examples. Nor was a model found 
among school districts contacted around the country. ORE staff were obliged to rely on their own 
acumen and experience in c aveloping a methodology which used educational effect measures and 
which accommodated the diverse array of programs in AISD. 

In addition to the methodological challenge, ORE staff were mindful of the political context. Reporting 
the cost-effectiveness of a variety of special programs, many of which had special interest groups 
ready to defend them against any negative evaluation, was a bold venture into previously avoided 
territory. Controversy over the methodology used and the ratings assigned to programs was expected. 
At the same time, overreliance by decison makers on the results of a new methodology, especially in 
making decisions about the future of programs, was a legitimate concern. 



MANDATE: 
Board requested ORE to indicate effect and cost 
in program effectiveness charts 

ORE'S RESPONSE: 

Retrospective look at 1991-92 evaluations 
including cost effectiveness 

ORE'S 1992-93 Agenda includes an evaluation plan 
to compare programs in terms of cost effectiveness 



METHOD 



Overview 



The following is a general overview of the method. A more detailed explanation is contained in the "Method" 
section of this paper. 



Following Henry Levin's definition of 
cost-effectiveness (Levin, 1983) cost- 
effectiveness is obtained by dividing 
cost by effect. 

cost 



Cost Effectiveness - effect 

The equation is very simple, but 
assigning values to the terms in the 
numerator and denominator is 
complex and can be controversial. 

Program costs are reported as 
budgeted amounts. Actual 
expenditures may vary. Some 
programs with relatively low costs 
may require substantial indirect 
resources for staff support, facilities, etc. 
of those programs could cost more if the 
expansion. 



DEFINITIONS 



Cost: Appropriation (Budget) 



Effect: 0- Achievement w**™<™m ; i 
Not dropping out , j 

i i 
i 

0* Other Indicators (Not used In formula) \ j 



Volunteers hold the costs down in some programs, but expansion 
pool of available volunteers is not large enough to accommodate 



Outcomes or Effect 

Program outcomes or effect are 
reported in terms of achievement 
gains, not dropping out, or other 
indicators. If available, norm- 
referenced test (NRT) scores were 
used. If the program focused on 
dropout prevention, then the dropout 
rate was used. This procedure seems 
straightforward, but an NRT is only 
one of many measures of student 
academic progress. Criterion- 
referenced tests, college entrance 
exams (SAT and ACT), grade-point 
average (GPA), and many other 
alternatives could be used. An NRT 
was chosen because it is our most 
reliable, broadest based, and most 
readily available measure. In order to 
compare cost-effectiveness across 
programs, a single effect measure is essential. 



EFFECT 



ACHIEVEMENT 



' I vs 

[ 

•:': : '|piftjict AvtfAQ* 
or 



NOT DROPPING OUT 



RATING 



OTHER MEASURES 



Survey* 

>* Staff v : 
Student*;; 

lAttendanoi v •./••:: 

•;l!HidpCr^ : : ; ;:-k ; : : : : :: !:x ; 



ERIC 



PEST COPY AVAILABLE 



When an NRT score was used, outcomes are reported as the achievement gain in Qrade equivalent (GE) months- 
above and beyond what the students would have gained without the program. A grade equivalent month is the 
amount of gain made on the Norm-referenced Assessment Program for Texas (NAPT) or the Iowa Tests of Basic 
Skills {ITBS) by an average student during one month of instruction. 



For programs for at-risk students, the 
dropout rate is clearly appropriate. 
Although these programs can certainly 
have benefits beyond just keeping 
students in school, the purpose of the 
calculation was to look simply at how 
much the program spent to keep one 
student from dropping out. In other 
words, if the student population 
served typically has 20 dropouts 
annually, and among the program 
students only 15 dropped out, then 
the program is credited with keeping 
five in school. This can make the cost 
per student kept in schoc 1 high, 
because 20 at-risk students may have 
to be served to net one dropout kept 
in school. 



COSTTEFFECT { C/E ) 
Cost/ Achievement Gain 



or 



Cost / %of potential dropouts 
staying in school 



Programs for which no norm-referenced test or dropout data were available were rated based upon other 
information from their evaluation reports. 

Cost-effectiveness 

Outcomes are divided into the cost of the program per student to give the cost to produce one month of 
achievement gain, or into the total program cost to calculate the cost to keep one potential dropout in school. 
A caution to the reader is that a program may not be able to produce twice the effect for twice the cost. We 
do not know what relationships would exist if we spent more or less money on a program. However, this cost- 
effectiveness number does tell us what we did spend for the amount of benefit realized. 

Cost-effectiveness could not be shown for some programs, either because they had no positive effect or 
because their impact was actually negative. 

Data Sources 

The data reported in this paper derive from cost-effectiveness analyses performed by ORE during the 1992-93 
school y.*5ar on 1991-92 AISD programs. Cost information was obtained from District files of financial 
information. Effectiveness information was developed from many sources but of chief interest here are 
achievement and dropout data, which were obtained from disk files maintained by ORE on the District's IBM 
4381 mainframe. 




Analyses 



The method used to calculate cost-effectiveness is detailed in the "Method" section of this paper. A few 
re/narks here about the major components of the analyses performed will be helpful, however. 

Type of Analysis 

Although it should be evident from the terminology, it was sometimes necessary to distinguish for ORE staff and 
others that the type of cost analysis we were pursuing was cost-effectiveness analysis, where the denominator 
in the equation is a measure of effectiveness expressed in nonmonetary terms, rather than cost-benefit analysis, 
where the denominator is benefit expressed in dollars. Cost-effectiveness is often used as a general term to 
connote any type of cost analysis, even by some educational researchers, but different cost analyses, while they 
have cost as a common term in the numerator, have different terms in the denominator. If cost-benefit ratios 
were desired, it would be necessary to assign monetary benefits to program outcomes; then costs expressed 
in dollars would be divided by benefits expressed in dollars. 

Cost 

In previous years, ORE had grappled with the issue of how to "cost" programs (see Wilkinson, 1985) and had 
arrived at several operational decisions. One of these was that a program's initial budget, or appropriation, 
should be used as the program's cost, rather than actual expenditures. The rationale for this decision was that 
while a program may not have expended its entire appropriation, the entire amount appropriated was reserved 
to it and was therefore not available to some other program. Imbedded in the appropriation, therefore, was a 
kind of "opportunity cost"; the amount of unexpended funds could have been spent by some other program if 
it had had the opportunity. 

Another important operational decision had to do with inequity of "charging" to a program the entire amount 
of its start-up costs in its first year of operation, especially when a large quantity of equipment was purchased, 
as in a technology program. If the program operated over a period of years, its average annual cost would be 
much less than its actual first-year cost. Although it could be argued that the difference between a program's 
first-year cost and subsequent years' costs was a reflection of reality, and that there was no guarantee that the 
program would continue to operate for the numbers of years projected, it seeded nonetheless an artificial 
distortion of the cost to run the program to include the start-up costs in one year's budget. Alternatives 
included prorating the start-up costs over the projected years of operation or depreciating capital outlays in 
second and successive years. For the purposes of these analyses, the decision was made to distinguish 
between "investment cost" and annual "operating cost," to report both, but to base cost calculations only on 
operating cost. 

While there may be more precise ways to determine cost-and generally accepted accounting practices for 
educational programs will probably be adopted in the future-ORE's emphasis in this study was on the effect side 
of the equation, and therefore the determination of costs was generally along the lines of previous work. 

Report On School Effectiveness (ROSE) 

In determining which effect measures to use, considerable attention was paid to the importance of distinguishing 
the effect of a special program from that of the general educational program. In other words, the question to 
be answered was, "What happened as a result of this program which would not have occurred without the 
program?" For example, it is not enough simply to measure the achievement gain made by a group of students 
in a special program; it is necessary to try to assess achievement gain above and beyond what the students 
would normally have gained without the program. The underlying concept is a comparison of actual outcome 
with some anticipated outcome given that the program has an effect. Whether that anticipated outcome is 
termed predicted, average, norm, target, or whatever, the difference between the actual and the standard 
aaainst which it is compared is logically attributable to the effect of the program. 
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To establish a standard against which to compare achievement outcomes, ORE made use of an analysis it had 
developed to measure the effectiveness of schools and later extended to programs. Both the Report On School 
Effectiveness (ROSE) and the Report On Program Effectiveness (ROPE) utilize regression analysis to predict 
student achievement, whether by all the students in a school or by students in a special program, and then to 
compare predicted achievement with actual achievement. The difference between predicted and actual 
achievement, expressed in grade equivalents (GE's), is the ROSE/ROPE residual. The ROSE residual was used 
as the achievement effect measure {denominator) for many of the cost-effectiveness analyses. See "Method" 
for more information about ROSE and about alternative achievement effect measures. 

Dropout Prediction 

Like the achievement effect measure, the dropout prevention effect measure was expressed as a residual, 
between the number of students in a program predicted to drop out and the actual number who dropped out. 
The "Method" and "Definitions" sections of this paper contain a detailed explanation. A great deal of discussion, 
continuing through the present, has been devoted to the method used to establish the predicted number of 
dropouts. The method is predicated on the longitudinal tracking of AISD dropouts and their associated risk 
factors. However, only academic risk factors are used to calculate the predicted number of dropouts. Other 
risk factors which are better predictors of dropping out need to be incorporated into the analyses. This issue 
will be discussed further in "Refinements to the Methodology in 1993-94." 

GENeric Evaluation SY Stem (GENESYS) 

Another important component in the cost-effectiveness analyses was ORE'S GENeric Evaluation SYStem 
(GENESYS). Developed to provide a wide range of evaluation information about a program, GENESYS relies on 
a large number of District-maintained computer files. GENESYS produces, among other things, ROSE and ROPE 
analyses. Most of the GE's used in calculations of achievement effect were obtained from GENESYS. GENESYS 
also produces statistics on the predicted and actual numbe" of dropouts. Tor more information about GENESYS, 
see "Method." See also Ligon snd Saenen (1 989) and Wilkinson and Spano (1 990). 

RESULTS 

Summary 

The major findings of the study were: 

1. ORE reviewed 85 1991-92 programs or program components. Cost-effectiveness was calculated for 
18 programs using an achievement effect measure and for 16 programs using a dropout prevention 
effect measure. An additional 14 dropout prevention programs were rated on effectiveness, although 
cost information could not be obtained. Another 37 programs were rated on effectiveness based on 
other evaluation information. 

2. Most programs evaluated in 1S91-92 in AISD were rated as effective. Approximately 21% of the 
ratings were based on achievement, 35% were based on the number of students not dropping out, and 
44% were based on other evaluation findings. 

3. In genera!, the programs showing the highest achievement gains for students tend to be programs that 
offer students enriching experiences in addition to the reaular curriculum. Most of these programs have 
a relatively high initial cost, but once the program is in place the cost for the per-pupil gain is relatively 
low. 

4. A common feature among successful dropout prevention programs is that they provide students with 
individual attention or the possibility of flexibility in class schedules and enrichment activities. Many of 
these programs are dependent on the use of volunteers or mentors, so they would not be as cost- 
effective if the District were to purchase the same services. 
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Discussion of Findings 



Most programs evaluated in 1991-92 in AISD were rated as effective. Approximately 21 % of the ratings are 
based on achievement, 35% are based on the number of students not dropping out, and 44% are based on 
other evaluation findings. 



Achievement Gains 

In general, the programs showing the 
highest achievement gains for 
students served tend to be programs 
that offer students enriching 
experiences in addition to the regular 
curriculum. Most of these programs 
have a relatively high initial cost. But 
once the program is in place, the gain 
for the par-pupil cost is relatively low. 




Successful Program 

^Computer Lab at Read 
^Science Academy 
Tfir Liberal Arts Academy 
Sfe-Ch. 1 Supplementary 
^Secondary Honors 
^Gifted & Talented (Ben 
Kealing 



• ; :v, ; ,v>: ; :v;v; : .;<. : X;: : ,;<.;<,;,;.<;.!.;,;.;.;-;< 

; ■ 



Effect C/E Index 



(Gain) 


($) 


9.5 


6 


6.4 


210 


3.4 


443 


3.3 


530 


2.3 


0 


1.7 


51 


1.3 


410 



Dropout Prevention 

A common feature among successful 
dropout prevention programs is that 
they provide students with individual 
attention or the possibility of flexibility 
in class schedules and enrichment 
activities. Many of these programs 
are dependent on the use of 
volunteers or mentors. The cost 
reported for these programs does not 
reflect the in-kinC contribution of 
volunteers. 



WHAT WORKS 



Successful Dropout Prevention Programs 



^-Johnston Tech. Lab 
X Block Q rog. at Travis 
^ Title VII Newcomers 
&PEAK 

Block Prog, at Lanier 
$ Adopt a student 



Served 

678 
175 
104 
163 
144 
31 



%Who 
Stayed 

29 
10 
7 
7 
5 
3 



The following sections contain charts showing the (1) cost-effectiveness of 1991-92 programs based on an 
achievement measure, (2) cost-effectiveness of 1 991-92 programs based on a dropout prevention measure, and 
(3) effectiveness of 1991-92 programs based on other indicators. Each section is prefaced by an example 
showing how cost-effectiveness was calculated for the programs in the section. 

6 
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COST-EFFECTIVENESS OF 1991-92 PROGRAMS BASED ON AN ACHIEVEMENT MEASURE 



Example 



PROGRAM 


RATING 


ALLOCATION 
{COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 
1 MONTH GAIN 
(COST/EFFECT) 


Elementary Computer Lab 
Funding Source: Local 
Grades: 5 

Level of Service: 45-85 hours/week 


+ 


$15,925 


264 


$60 


R: 1 1 .0 
M: 8.0 
Avg.: 9.5 


$6 



Elementary Computer Lab, 1991-92 - Grades: 5 - Level of Service: 45-85 minutes/week 



Rating: + 
Cost: $15,925 

Number of Students Served: 264 
Cost Per Student: $60 [$15,925/264 = $60] 
Effect: R: 11.0 M: 8.0 Avg. = 9.5 
Cost/Effect: $6 [$60/9.5 = $6.32] 

What this means is that it costs $6 per year per Elementary Computer Lab student attending 
the computer lab to attain one month's achievement gain above that the student would 
normally have achieved as the result of the regular instructional program. 
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Cost-Effectiveness of 1991-92 Programs Based on an Achievement Measure 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Biiingual/ESL 
Funding Source: Local 
Grades: K-12 
Level of Service: Varies 


0 


$831,524 


6,108 


$136 


R: 0.0 
M: 0.2 
Avg.: 0.1 




Chapter 1 School Projects (all 
students) 

Funding Source: External 
Grades: K-6 

Level of Service: All day/all year 


0 


$1,787,173 


6,328 


$282 


R: -0.8 
M: N/A 
Avg.: N/A 




Chapter 1 Schoolwide Projects 
(low achievers) 

Funding Source: External 

Grades: K-6 

Level of Service: Ail day/all year 


+ 


$245,683 


428 


$574 


R: 1.5 
M: N/A 
Avq.: N/A 


$383 


Chapter 1 Supplementary Instruction 
(low achievers) 

Funding Source: External 

Grades: 1-6 

Level Of Service: 30 min. per day/aJl year 


+ 


$785,538 


1,482 


$530 


R: 3,3 
M: N/A 
Avg.: N/A 


$160 



• Participants 



Rating it expreaaed at contributing to any of tha 6 AISD 
■tratagic objectivat. 

+ Positive, naedt to be kept and expanded 

0 Not *ignific*nt, naedt to be improved and modified 

Negative, naedt major modification or replacement 
Blenk Unknown, mey have poertive or negative impact on 

other indicaton; however, impact on the five AISD 

ttretegtc object] vea ia unknown. 



R:~ Reading 
M:> Mathamatice 
Avg.: « Average 
(sometimes average is 
weighted by number of 
students) 



Coat ia the expenae over the regular Diatrict per atudent 
expenditure of about 12,000. 

0 No cogt or minimal coat 

1 indirect com and overhead, but no eeperate budget 
II Some direct cats, but under $600 par ttudant 

III Mefor direct co%t% for teachera. ttaff, and/or 
equipment in the range of 1500 per atudent 
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Cost-Effectiveness of 1991-92 Programs Based on an Achievement Measure 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Chapter 2 Elementary Computer Lab 
at Read 

Funding Source: External 
Grades: 5-6 

Level Of Service: 45-85 min. /week/ell yeer 


+ 


$15,925 

Investment Cost 
(est.) $16,000 for 
software and & 
Apple computers 
bought in 
1989-90 


264 


$60 


R: 1 1 .0 
M: 8.0 
Avg.: 9.5 


$6 


DFS Student Alcohol and Drug 
Education Prevention Program 

Funding Source: External 

Grades: 5-12 

Level of Service: N/A 


+ 


$149,009 


1,711 


$87 


R: -0.04 
M: -0.14 
Avg.: -0.09 

Positive student 
•urv»y r—utin, 
lower retention, 
& lower dropout 
rata 




Elementary Computer Lab at Blanton 
Funding Source: External 
Grades: 2-5 

Level of Service: 20-30 min./d«y/«u year 


0 


$56,522 

Investment Cost 
(est.) $100,000 


402 


$141 


R: 0.7 
M: -0.6 
Avg.: 0.05 




Gifted & Talented (Elementary) 
Funding Source: Local 
Grades: K-6 
Level of Service: Varies 


+ 


$342,156 


3,922 


$87 


R: 2.0 
M: 1.4 
Avg.: 1.7 


$51 


Gifted & Talented (Secondary) 
Funding Source: Local 
Grades: 6-1 1 

Level of Service: 1 or more honors coureee 


+ . 


$0 


8 f 321 


$0 


R: 2.7 
M: 2.0 
Avg.: 2.3 




Kealing Magnet 
Funding Source: Local 
Grades: 7-8 

Level of Service: All year 


+ 


$221,491 

Investment Cost 
(est.) $10,000 


432 


$513 


R: 2.0 
M: 0.5 
Avg.: 1.25 


$410 
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Cost-Effectiveness of 1991-92 Programs Based on an Achievement Measure 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
COST/EFFECT) 


Liberal Arts Academy 
Funding Source: Local 
Grades: 9-12 
Level of Service: All day 


+ 


$450,296 

Investment Cost 
(est.) $173,833 


276 


$1,632 


R: 5.4 
M: 1.3 
Avg.: 3.4 


$480 


MegaSkills Parent Training 
Funding Source: External 
Grades: 2-6 

Level of Service: 5-8 workshops 


+ , 


$75,630 

($40,650 provided by 
DFS Grant 121,980 
Chapter 1, $13,000 
itm bos in sea es) 


1,196 


$63 


H: U.Ub 
M: 0.10 
Avg : 0.08 

Highest etten- 
dencs, lower 
discipline, & 
lower retention 
rate 




Priority Schools Overall (Low 
achievers) 

Funding Source: External & Local 

Grades: Pre-K through 6 

Level of Service: All day/all year 


+ 


$5,227,579 


7,557 


$692 


R: 3.4 
M: N/A 
Avg.: N/A 


$204 


Project A+ Elementary Technology 
Demonstration Schools: Andrews 

Funding Source: External 

Grades: K-5 

Level of Service: All day/all year 


0 


$63,253 

$1,100,956 
Investment cost 
for hardware, 
software, and 
wiring 


843 


$75 


R: 0.0 
M: -0.25 
Avg.: -0.13 




Project A + Elementary Technology 
Demonstration Schools: Patton 

Funding Source: External 

Grades: K-5 

Level of Service: All day/all year 


0 


$63,253 

$1 f 354,320 
Investment cost 
for hardware, 
software, and 
wiring 


1,037 


$61 


R: -0.5 
M: 0.0 
Avg.: -0.25 





• Participants 



Rating it expressed ■■ contributing to any of the 5 AISD 
strategic objectives. 



+ 
0 



Blank 



o 

ERLC 



Positive, needs to be kept and expanded 
Not tignifkxnt, need a to be improved and modified 
Negative, needs major modification or replacement 
Unknown, may have positive or negative Impact on 
other indicators; however, impact on the five AISD 
strategic objectives is unknown. 



R:~ Reading 

M: = Mathematics 

Avg.: a Average 

(sometimes average is 

weighted by number of 

students) 



Coet is the expense over the reguter District per student 
expenditure of ebout I2.0O0. 

0 No cost or minims! cost 

♦ indirect co$t$ end overhead, but no seperete budget 
II Some direct co*tt, but under 1600 per etudent 
III Mejor direct costs for teachers, stetf. end/or 
equipment in the range of 1500 per student 
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Cost-Effectiveness of 1991-92 Programs Based on an Achievement Measure 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Project A + Elementary Technology 
Demonstration Schools: Langford 

Funding Source: External 

Grades: K-5 

Level of Service: All day/all year 


0 


$53,744 


574 


$94 


R: 2.0 
M: -0.25 
Avg.:0.875 




$749,642 
Investment cost 
for hardware, 
software, and 
wiring 


Project A + Elementary Technology 
Demonstration Schools: Galindo 

Funding Source: External 

Grades: K-5 

Level of Service: All day/all year 


0 


$44 r 235 


751 


$59 


R: 0.0 
M: 1.25 
Avg.:0.625 




$246,000 
Investment cost 
for hardware, 
software, and 
wiring 


Science Academy 
Funding Source: Local 
Grades: 9-12 
Level of Service: All year 


+ 


$815,604 


608 


$1,341 


R: 8.3 
M: 4.5 
Avg.: 6.4 


$210 


$513,711 
Investment cost, 
local and grant 
sources 
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COST-EFFECTIVENESS OF 1991-92 PROGRAMS BASED ON A DROPOUT PREVENTION MEASURE 



Example 



PROGRAM 


RATING 


ALLOCATION 

iCOST) 


NUMBER 
OF 

STUDENTS 
SERVED 


COST 
PER 

STUDENT 


NUMBER OF 
DROPOUTS 

Predicted Obtained 


PREDICTED 
DROPOUTS 
WHO 

STAYED IN 
SCHOOL 
(EFFECT) 
# % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


Dropout Prevention Program 
Funding Source: External 
Gradee: 9-12 


+ 


♦ 100,000 


140 


$714 


12 3 


9 75 


$11,111 



Dropout Prevention Program, 1991-92 - Grades: 9-1 1 - Level of Service: 3 hours/day 
Rating: + 
Cost: $100,000 

Number of Students Served: 1 40 

Cost Per Student: $714 [$100,000/140 = $714.28 = $714 roundec! 
Effect: 9 

[Predicted 9.1 %, Obtained 2.9% 

2.9/9.1 = .31868 = 32% of predicted rate, or 68% "saved" from looping o-jc 
.091 x 140 students = 12.74 = 12 rounded = 12 students predicted to drop out 
.68 x 12 students = 8.84 = 9 rounded = 9 students "saved"] 
Cost/Effect: $11,111 [$100,000/9 = $1 1 ,1 1 1 .II = $1 1 ,1 1 1 rounded] 

What this means is that it costs $11,111 for each student "saved" from dropping out by the Dropout 
Prevention Program who would otherwise have been expected to drop out of school. 
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Cost-Effectiveness of 1991-92 Programs Based on a Dropout Prevention Measure 



rnUvJnMW 




ALLOCATION 
{COST) 


NUMBER 
OF 

STUDENTS* 

otnVtu 


COST 
PER 

STUDENT 


NUMBER OF DROPOUTS 


PREDICTED 
DROPOUTS 
WHO STAYED 
IN SCHOOL 
(EFFECT) 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


Adopt A Student at LBJ 
Funding Source: Local 
Grades: 9-12 


+ 




31 


$0 


3 0 


3 100 


$0 


Alternative Learning Center (ALC) 
Funding Source: Local and External 
Grades: 9-12 


+ 


$429,760 


104 


$4,132 


104 # * 30 


74 71 


$5,804 


Alternative Learning Center (ALC) 
Funding Source: Local and External 
Grades: 7-8 


+ 


$628,111 


152 


$4,13^ 


152" 31 


121 80 


$5,191 


Block Program-Crockett (Success) 
Funding Source: Local 
Grades: 9-12 


+ 




79 




2 1 


1 50 




Block Program-Lanier (Connections) 
Funding Source: Local 
Grades: 9-12 


+ 




*1 14 




5 0 


5 100 




Block Program-Reagan 
Funding Source: Local 


+ 




45 




2 1 


1 50 




Block Program-Travis (Excel) 
Funding Source: Local 
Grades: 9-12 


+ 




175 


0 


14 4 


10 71 





• Participants 



* All students in program are at risk by definition. 



Rating is expremed ■■ contributing to any of the 5 A!SD 
strategic objective!. 

+ Positive. needi to be kept and expanded 

O Not B/gnrffcsnt, needi to be improved and modified 

Negmtive. needi major modification or replacement 
Blank Unknown, may have poiitive or negative impact on 

other indicaton; however, impact on the five AISD 

strategic objectives is unknown. 
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Coit is the expenie over the regular Diitrict per student 
expenditure of about $2,000. 

0 No cOBt or minimal coit 

% indirect costs and overhead, but no separate budget 
$♦ Some direct cotts, but undar $600 per itudont 
$$$ Msfor direct com for teachers, staff, and/or 
equipment in the range of $500 per student 
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' Cost-Effectiveness of 1991-92 Programs Based on a Dropout Prevention Measure 



PROGRAM 


RATING 


ALLOCATION 
<COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


NUMBER OF DROPOUTS 
Predicted Obtained 


PREDICTED 
DROPOUTS 
WHO STAYED 
IN SCHOOL 
(EFFECT! 
# % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


Communities in School (CIS) at 
Fulmore 

Funding Source: External 

firaHpe* 7-ft 


+ 




52 




2 0 


2 100 




CIS at Pearce 

Funding Source: External 


0 




18 




1 1 


0 0 




CIS at Porter 

Funding Source: External 

Grades* 7-fl 


+ 




41 




1 0 


1 100 




CIS at Robbins 

Funding Source: External 

Gradps- 9-1 2 


+ 




79 


0 


79" 15 


64 81 




CIS at Travis 

Funding Source: External 

Grades: 9-12 


+ 




40 


0 


3 1 


2 33 




CVAE 

Funding Source: Local 
Grades* 9-1 2 


- 




361 


0 


39 47 


-8 -21 




Evening School 

Funding Source: Local and External 
Grades: 9-1 2 


+ 


$329,346 


300 


$1,098 


300" 60 


240 80 


$1,3^2 


Hispanic Student Scholarship 
Initiative (HSSI) at Martin 

Funding Source: External 

Grades: 7-8 


0 


$0 


31 


$0 


0 o 






Johnston Technology Learning Ctr. 
Funding Source: External 
Grades: 9-12 


+ 


$56,838 


678 


$84 


34 5 


29 85 


$1,672 



Cost-Effectiveness of 1991-92 Programs Based on a Dropout Prevention Measure 



PROGRAM 


RATING 


ALLOCATION 
1COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


NUMBER OF DROPOUTS 
Predicted Obtained 


PREDICTED 
DROPOUTS 
WHO STAYED 
IN SCHOOL 
{EFFECT) 
# % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT* 


Jtirnpstart (McCallum) 
Funding Source: Local 


0 


$0 


11 


0 


0 0 






Mentor 

Funding Source: External 


+ 




41 




2 0 


2 100 




Mentor 

Funding Source: External 


+ 




92 




1 0 


1 100 




Newcomers Program (Title VII) 
Funding Source: External 

UfdUcb. 1 Z. 


+ 


$140,000 


104 


$1,346 


104" 3 


101 97 


$1,386 


Peer Assistance Leadership (PAL) 
Funding Source: External 
Grades: 9-12 


+ 


$13,162 


48 


$274 


3 0 


3 100 


$4,387 


Peer Assistance Leadership (PAL) 
Funding Source: External 
Grades: 7-8 


+ 


$46,886 


171 


$274 


1 0 


1 100 


$46,886 



Participants 



* All students in program are at risk by definition. 



Rating it expressed m contributing to any of the 5 AISD 
etretegic objectives. 



+ 
O 

Blank 



ER?C 



Positive, neede \o be kept end exoended 
Not significant, neede to be improved end modified 
Negative, neede major modification or replace ment 
Unknown, may have positive or negative impact on 
other indicatore; however, impact on the five AISD 
at rat eg ic objectives it unknown. 
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Coat ia the expense over the regular District per student 
expenditure of about $2,000. 

0 No cost or minimal coat 

$ indirect costs and overhead, but no asperate budget 
$$ Some direct costs, but under $600 per atudant 
$$$ Me/or direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 
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' Cost-Effectiveness of 1991-92 Programs Based on a Dropout Prevention Measure 



PROGRAM 


RATING 


ALLOCATION 
<COST) 


NUMBER 
OF 

STUDENTS - 
SERVED 


COST 
PER 

STUDENT 


NUMBER OF DROPOUTS 
Pr«tict«J Obtained 


PREDICTED 
DROPOUTS 
WHO STAYED 
IN SCHOOL 
(EFFECT) 
# % 


COST PER 

STUDENT 

KEPTJN 

SCHOOL 

(COST/ 

EFFECT) 


Practical Effective Application of 
Knowledge (PEAK) Program (Austin, 
McCallum) 

Funding Source: Local 
Grades: 9-12 


+ 


90 


34 


9U 


/ u 


o 1 nn 


vU 


Practical Effective Application of 
Knowledge (PEAK) Program - Spring 
1992 

Funding Source: Local 
Grades: 9-12 


+ 


$0 


129 


$0 


5 u 


O 1UU 




Project MAN (Men Act Now) at LBJ 
Funding Source: External 
Grades: 9-12 


+ 


$0 


28 


0 


1 0 


1 100 


$0 


Reading Tutor at Austin Fall 1991 
Funding Source: Local 
Grades: 9-12 


0 




25 




1 1 


0 0 




Reading Tutor at Austin Spring 1 992 
Funding Source: Local 
Grades: 9-1 2 


+ 




21 




1 0 


1 100 




Robbins 

Funding Source: Local and External 
Grades: 9-12 


+ 


$1,333,238 


536 


$2,487 


536* 94 


442 82 


$3,016 


Texas Associates of Minority 
Engineers (TAME) Club at Bowie 

Funding Source: Local 

Grades: 9-1 2 


0 


$210 


53 


$4 


1 1 


0 0 




Zenith at Evening School 
Funding Source: Local 
Grades: 9-1 2 


+ 


$131,990 


323 


$409 


323* 16 


307 95 


$430 
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EFFECTIVENESS OF 1991-92 PROGRAMS BASED ON OTHER INDICATORS 

EXAMPLE 



PROGRAM 


ALLOCATION 
{COST) 


NUMBER OF 

STUDENTS 

SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 

STUDENT 

FOR 1 

MONTH 

GAIN 

(COST/ 

EFFECT) 


RATING 


Supplementary Instructional Program 
Funding Source: External 
Grades: K-6 

Level of Service: 1-2 hours per w©©k 


$144,200 


128 


$1,127 


Rating based 
on program 
meeting its 
goals 




+ 



Supplementary Instructional Program, 1991-92 - Grades: K-6 - Level of Service: 1 -2 hours/week 
ftating: + 
Cost: $144,200 

Number of Students Served: 1 28 

Cost Per Student: $1,1 27 [$144,200/1 28 = $1,126.56 = $1 ,1 27 rounded] 
Effect: No NAPT/ITBS or dropout data available 

[Because no appropriate achievement test data or dropout prevention data were available, the rating of this 
program is based on other indicators, in this case, a measure of the extent to which the program is meeting 
its goals.] 

Cost/Effect: Cannot be calculated 

[In the absence of an effect measure comparable to that used with other programs by which to divided the 
cost (i.e., 

the denominator), a cost-effectiveness ratio cannot be calculated.] 

What this means is that it costs $1 ,127 per year per Supplementary Instructional Program student to attain 
progress toward the program's goals, but the cost-effectiveness of the program relative to other programs in 
terms of its effect 

on student achievement or dropout prevention cannot be determined. 
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Effectiveness of 1991-92 Programs Based on Other Indicators 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Chapter 1 Migrant Supplementary 
Instruction 

Fundino Source: External 
Grades: K-12 

Level of Service: One to two hour* per 

week/fufi veer 


+ 


$144,002 


128 


$1,125 


Rating based 
on program 
meeting its 
goals 




Chapter 1 Neglected or 
Delinquent Institutions 

Funding Source: External 

Grades: 1-12 

Level of Service: Varied 


+ 


$75,498 


1,054 


$72 


no Lilly UddCU 

on 

institutions 
meeting tiieir 
goals for the 
year 




Chapter 1 Nonpublic Schools 
Funding Source: External 
Grades: 1-7 

Level of Service: 30 min./day/all year 


+ 


$16,377 


22 


$744 


Rating oased 
on program 
meeting its 
goals 




Chapter 2 Academic Decathlon 
Funding Source: External 
Grades: 11-12 

Level of Service: Varied by school 


+ 


$38,609 


76 


$508 


Rating based 
on employee 
survey 
results 





* Participants 



Biting it axpreieed •■ contributing to my of the 5 AISO 
■tritegk: objectives. 



+ 
0 



Blink 



Positive, needi to be kept md expended 
Not significant, needi to be improved end modified 
Neastive. needi mejor modification or replacement 
Unknown, may have positive or negative impict on 
other indicitori; however, impact on the five AISD 
■tritegic objective! ii unknown. 



R: = Reading 
M:*= Mathematics 
A v g . : = Average 

(sometimes overage is 
weighted by number of 
students) 



Co it tt the expenie over the regulir Oiitrict per itudent 
expand rturo of about $2,000. 

0 No cost or minimal coit 

% Indirect costs and overhead, but no lepinte budget 
%% Some direct costs, but under 4bO0 per itudent 
Mt Msjor direct costs for teachers, itiff, and/or 
equipment in the range of $500 per itudent 



ERIC 
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Effectiveness of 1991*92 Programs Based on Other Indicators 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PGR 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Chapter 2 Elementary Computer Lab 
at Blackshear 

Funding Source: Externa! 

Grades: K-1 and ED students in 
grades 1-3 

Level of Service: 46-60 mmut««/d«y/«H 
year 


+ 


$17,191 

investment Cost 
(est.) $16,000 for 
hardware and 
software plus 
$1,000 annually 
for consumable 
materials 


123 


So 


Rating based 
on employee 
survey 
results 




Chapter 2 Extracurricular 
Transportation 

Funding Source: External & Local 
Grades: 6-1 2 

Level of Service: As requested 


+ 


{Chapter 2 $96,418; 
Local 198,296) 


0*tU 


£ 1 fl9 

v 1 OZ. 


Rating based 
on Student 

dilU 

Employee 

survey 

results 




Chapter 2 Library Resources 
Funding Source: External 
Grades: Pre-K through 12 
Level of Service: N/A 


+ 


$40,310 


66,705 


$1 


naung oaseo 
on employee 
survey 
results 




Chapter 2 Middle School Homeroom 
Training 

Funding Source: External 
Grades: 6-3 

Level of Service: None 




$3,379 


0 




Kin traininn 

held 




Chapter 2 Multicultural/Special 
Purpose Buses 

Funding Source: External 

Grades: Pre-K through 12 

Level of Service: As requested/all year 


+ 


$12,000 


9,450 


$1 


Rating based 

and bus user 

survey 

results 




Chapter 2 Private Schools 
Funding Source: External 
Grades: Pre-K through 12 
Level of Service: N/A 


+ 


$21,419 


2,766 


$8 


Rating based 
on private 
school 
survey 
results 
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Effectiveness of 1991-92 Programs Based on Other indicators 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS • 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Chapter 2 Secondary Library 
Technology Support 

Funding Source: External 

Grades: 6-12 

Level of Service: N/A 


+ 


$19,833 


12,032 


$2 


Rating based 
on purchases 
made 




Chapter 2 Spanish Academy 
Funding Source: External 

Grades: Any AISD staff m«rnb#f is eligible to 
participate 

Level of Service: N/A 


+ 


$38,774 


213 
Staff 


$182 


Rating based 
on 

participant 
survey 




Chapter 2 Support for Restructured 
Robbins 

Funding Source: External 

Grades: 9-12 

Level of Service: N/A 


+ 


$7,000 


361 


$19 


Rating based 
on principal 
interview 




Chapter 2 Technology for Access to 
Problem Solving 

Funding Source: External 

Grades: 8 

Level Of Service: Calculator* & training one 

time 


+ 


$0 

(Calculators 
provided by TEA) 


4,324 


$0 


Rating based 
on employee 
survey 
results 





* Participants 



Rating is expressed ei contributing to eny of the 6 AISD 
strategic objectives. 

+ Positive, needs to be kept end expended 
O Not significmnt, needs to be improved end modified 
Negative, needs major modification or replacement 
Blank Unknown, may have positive or neg stive impact on 
other indicetors; however, impact on the five AISD 
strategic objectives is unknown. 



R:*= Reading 
M:* Mathematics 
A v g . : « Avingi 

(sometimes average is 
weighted by number of 
students) 



Cost Is the expense over the regular District per student 
expenditure of about $2,000. 

0 No cost or minimal cost 

% Indirect costs and overhead, but no separate budget 
% % Some direct costs, but under 1600 per student 
$$$ Ms/or direct costs for teachers, staff, and/or 
equipment in the range of $500 per student 



ERIC 
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Effectiveness of 1991-92 Programs Based on Other Indicators 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FORI 
MONTH GAIN 
(COST/EFFECT) 


Chapter 2 Technology Learning Center 
at Johnston High 

Funding Source: External 

Grades: 9-12 

Level of Service: At r#qimt#d by 

damnrczm t*achar* for 

onrichrnant activitrM 


0 


$56,838 

mvestmeni uost 
(est,) $110,000 
for 25 station 
integrated 
Learning System 
<ILS) 


1,552 


$37 


Rating based 

UN sZHi^MUycc 

survey 
results 




DFS All Well Health Services 
Funding Source: External 
Grades: K-12 

Level of Service: Teachers Conference 


+ 


$3,000 


10 teachers 


$300 


Rating based 
on staff 
survey 
results 




DFS Conflict Resolution Project 

Funding Source: External 

Grades: 9-12 & staff 

Level of Service: 3 maatir>oa/month/aii y»w 


+ 


$55,147 


86 


$641 


Rating based 
on survey 
results 




DFS Drug Abuse Resistance Education 
Funding Source: External 
Grades: 5 & 7 

Level of Service: 5 hm./wi«fc for 17 & 

10 weoki respectively 


+ 


$686,110 

($594,302 provided 
by APD; $46,608 DFS 
grant, $30,000 
{undrawing, 1 16,000 
private contribution) 


10,023 


$4 


Rating based 
on survey 
results 




DFS Elementary Curriculum 
Funding Source: External 
Grades: PK-8 
Level of Service: N/A 




$40,886 


38,346 


$1 


Insufficient 
Information 




DFS Office of Student Intervention 
Services 

Funding Source: External 
Grades: K-12 

Level of Service: 1 play *t each of 27 
camputM 


+ 


$22,326 


5,560 


$4 


Rating based 
on survey 
results 


Not 
available 



Effectiveness of 1991-92 Programs Based on Other Indicators 



PROGRAM 


RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS' 
SERVED 


COST 
PER 

STUDENT 


EFFECT 
(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


DFS Parent Involvement 
Funding Source: External 
Grades: Adult 

Level of Service: 5 workshops 




$5,060 


202 
Parents 


$25 


No 

assessment 
conducted 




DFS Private Schools 
Funding Source: External 
Grades: PK-12 

Levci of Service: Varied by school 


+ 


$10,713 


1,717 


$6 


Rating based 
on staff 
survey 
results 




DFS Read Pilot 

Funding Source: External 

Grades: 5-6 

Level of Service: 1 5-day workshop 




$1,357 


264 


$5 


Insufficient 
Information 




DFS Peer Assistance Leadership 
Funding Source: External 
Grades: K-1 2 

Level of Service: 30-35 minutse/week/eii 

year 


+ 


$60,050 


1,509 


$40 


Rating based 
on survey 
results 





* Participants 



Rating it expressed as contributing to sny of the 6 AISD 
strategic objectives. 



+ 

O 



Blsnk 



Positive, needs to be kept snd expsrxled 
Not significsnt, needs to be improved and modified 
Negative, needs msjor modification or replacement 
Unknown, may have positive or negative impact on 
other indicators; however, imptct on the f've AISD 
strstogic objectives le unknown. 



R:* Reeding 
M:~ Mathematics 
A v g . : = Average 

(sometimes overage is 
weighted by number of 
students) 



Coat is the expense over the reguler District per student 
expenditure of about 12,000. 

0 No cost or minimal cost 

1 indirect costs and overhesd, but no separate budget 
tt Some direct costs, but under 1500 per student 

M$ Me/or direct costs for teachers, staff, and/or 
equipment in the range of *B0O per student 
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Effectiveness of 1991-92 Programs Based on Other Indicators 



PROGRAM 


RATING 


. I 
I 

ALLOCATION 
(COST) 


NUMBER 
DF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


4 

\ 

EFFECT 
(in months) 


:ost PER 

STUDENT FOR 1 
MONTH GAIN 
[COST/EFFECT) 


Full-Day Prekindergarten 
Funding Source: External 
Grades: Pre-K 

Level of Service: FuM-day c1m«m all yaar 


+ 


$1,291,422 


1,787 


$723 


Rating based 
on previous 
years' test 
results 




National Science Foundation 
Curriculum Development 

Funding Source: External 

Grades: K-12 

Level of Service: Varies 


+ 


$12,000 


20 
Teachers 


$600/ 
teacher 


Rating based 
on teacher 
survey 




National Science Foundation Private 
Sector Involvement 

Funding Source: External 

Grades: 9-12 

Level of Service: 


+ 






Insufficient 
Information 


Rating based 
on Director 
survey 




National Science Foundation Staff 
Development 

Funding Source: External 

Grades: 9-12 

Level of Service: 


+ 


$20,000 


80 
Teachers 


$250/ 
teacher 


Rating based 
on teacher 
survey 




Pregnancy, Education, and Parenting 
Funding Source: External 
Grades: 8-12 
Level of Service: Varies 




$120,000 


79 


$1,519 


Too few 
students per 
grade for 
analysis 




Project A + School Based 
Improvement - Phase 2 (Includes 
Phase I Schools) 

Funding Source: Local 

Grades: K-12 

Level of Service: All year 


+ 


$108,398 


24,489 


$4 


Rating based 
on programs 
initial 

implemen- 
tation goals 
being mfet 
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Effectiveness of 1991-92 Programs Based on Other Indicators 



PROGRAM 


RATING 


• 

ALLOCATION 
(COST) 


MUMBER | 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 

{in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Special Education 

Funding Source: Local/External 

Grades: K-12 

Level of Service: AH year 




$36,003,493 


9,129 


$3,944 






Title li Elementary Mathematics 
Funding Source: External 
Grades: K-5 

Level of Service: Varies 


+ 


$34,937 


61 


$573 


Rating based 
on 

participant 
survey 




Title II Elementary Science 
Funding Source: External 
Grades: K-5 

Level of Service: Varies 


+ 


$34,557 


122 


$283 


Rating based 
on 

participant 
survey 




Title II Gifted/Talented 
Funding Source: External 
Grades: K-5 

Level of Service: Varies 


+ 


$15,887 


308 


$52 


Rating based 
on 

participant 
survey 




Title II Secondary Mathematics 
Funding Source: External 
Grades: 6-12 
Level'of Service: Varies 


+ 


$41,082 


116 


$354 


Rating based 
on 

participant 
survey 





* Participants 



Rating it expressed ss contributing to eny of the 5 AISD 
strategic objectives. 

+ Positive, needs to be kept end expsnded 

0 Not significant, needs to be improved end modified 

Negative, needs mejor mod if ic it ion or re piece me nt 
Blank Unknown, mey neve positive or negetive impact on 

other Indicators; however, Impect on the five AISD 

strategic objectives is unknown. 



R:« Reading 
M:« Mathematics 
A v g . : = Average 

(sometimes average is 
weighted by number of 
students) 



Cost is the expense over the rsgulsr District per student 
expenditure of about $2,000. 

0 No cost or minimal cost 

$ Indirect costs end overhead, but no sepsrete budget 
*$ Some direct cottt, but under 1600 per student 
%%% Ms/or direct cotts for teechers, stsff, end/or 
equipment in the range of 1500 per student 
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Effectiveness of 1991-92 Programs Based on Other Indicators 





RATING 


ALLOCATION 
(COST) 


NUMBER 
OF 

STUDENTS* 
SERVED 


COST 
PER 

STUDENT 


EFFECT 

(in months) 


COST PER 
STUDENT FOR 1 
MONTH GAIN 
(COST/EFFECT) 


Title II Secondary Science 
Funding Source: External 
Grades: 6-12 
Level of Service: Varies 


+ 


$34,251 


59 


$581 


naiing oaseo 
on 

participant 
survey 




Title Vlt Secondary Bilingual 
Funding Source: External 
Grades: 9-1 1 
Level of Service: All year 




$140,000 


104 


$1,346 


Rating based 
on other 
school 
success 
indicators 
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CONCLUSIONS 



A year-long endeavor to combine measures of program effectiveness with fiscal resources expended 
led to the following conclusions: 

1. Information about the costs of special programs in our District, and probably in most others, 
is not easily obtainable at present. Costing a program is not a simple matter, but it is made 
more complicated in the absence of record keeping devoted to documenting program costs. 

2. Unless there are meaningful consequences, such as a public report, the priority placed by 
program staff on documenting program participation and cost information will be relatively low. 

3. For the purpose of calculating cost-effectiveness, standardized achievement test scores remain 
the best-i.e., the most reliable, broadest based, and most readily available-effect measure. 
Other proposed achievement measures, such as grades and to-be-developed performance 
assessment measures, do not serve the present purpose as well. Refinement of the dropout 
prevention measure used is needed. 

4. The methodology used in this study for calculating cost-effectiveness has great promise but 
also recognizable limitations. Further study and refinement of the methodology are needed to 
establish more confidence that it appropriately reflects how much learning is achieved for each 
dollar spent on special programs. 

IMPLICATIONS 

The combination of cost with effectiveness information enables the evaluation of programs in terms 
of their relative costs in meeting the same outcome criteria: improving student achievement or 
preventing students from dropping out of school. In other words, alternative programs can be 
evaluated on the basis of their costs for raising student test scores by a given amount or the cost for 
each potential dropout averted. Other success indicators notwithstanding, information about which 
programs provide the maximum effectiveness per level of cost or require the least cost per level of 
effectiveness will assist in decisions about which programs to keep and expand, which to modify, and 
which to discontinue. 

School board members and the general public increasingly request information about the cost and 
effectiveness of special programs about which budgetary decisions must be made. Actual cost- 
effectiveness information provides a dimension to the decision-making process which is often lacking. 
It is incumbent on evaluators, however, to assist decision makers such as school boards to understand 
the nature and limitations of the cost-effectiveness information being made available so that the 
information does not become the justification for the arbitrary sunsetting of programs because of 
budget reduction pressures rather than the programs' merit. Evaluators must resist being inveigled to 
draw conclusions which go further than their findings can sustain. At the same time, they must also 
resist criticism of their findings which arises more from political, self-serving motives than from a 
genuine concern with the quality of the data or the precision u the methodology. 

In sum, while decision makers and the general public favor eliminating ineffective and costly programs, 
assembling-then def ending-the information needed for "riding them off into the sunset" is a challenge 
both methodologically and politically. Evaluators who undertake to respond to requests for cost- 
effectiveness information should take care to prepare on both fronts. 
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REFINEMENTS TO THE METHODOLOGY IN 1993-94 



Since the 1 992-93 school year, when ORE first calculated cost-effectiveness, a few refinements have 
been made to the methodology. Chief among these has been to extend the calculations to include 
programs intended to prevent student alcohol and other drug abuse. 

Cost-Effectiveness of 1992-93 Programs Based on a Drug Preventi on Measure 

As with the other types of programs for which cost-effectiveness was calculated, the cost- 
effectiveness of drug prevention programs is calculated by dividing cost by a drug use prevention 
measure of effect. Cost was again appropriation. Effect was defined as not using drugs. 

Definition : Cost = appropriation (budget) 

Effect = not using drugs 

The drug prevention measure of affect was operationalized as the difference between the recent use 
of an illicit substance by program participants and by students in the District overall. 

Definition : Not using drugs = The difference between the recent use of an illicit 

substance by program participants and by students in the 
District overall 

The measure of students prevented from using drugs was based on self-reported use of alcohol, 
tobacco, and other illicit substances on the Student Alchol and Other Drug Use Survey, administered 
to AISD students in grades 4-1 2 in April 1 993. The survey included items about the students' recent 
use of illicit substances. For students in grades 4-5, recent use is defined as use within the past 
school year; recent use by students in grades 6-12 is defined as use within the past 30 days. 
Students were also asked about their participation in Drug-Free Schools programs. The rate of recent 
use of any illicit substance was calculated for program participants and for the District as a whole. 
The number of students prevented from alcohol and other drug use reflects the difference between 
recent use by program participants and overall recent use by the entire sample, multiplied by the total 
number of students served by the program. 

Cost-effectiveness was operationalized as cost of the program divided by drug prevention effect 
(average rate of drug use in the District minus the rate of use for program students times the number 
of students served by the program). 

Definitions : Cost/Effect = Cost for the program/drug prevention effect 

The cost-effectiveness ratio, expressed in dollars, which results from this division is a measure of the 
cost-effectiveness of a program, i.e., the amount of effect for monies expended, and because a 
common effect measure was used as the denominator among like programs, programs' cost- 
effectiveness can be compared. 

Definition : Cost/Effect = Cost-effectiveness ratio (in dollars) 

The following section contains a chart showing the cost-effectiveness of 1 992-93 programs based on 
a drug prevention measure. The section is prefaced by an example showing how cost-effectiveness 
was calculated. 
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COST-EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON 
A DRUG PREVENTION MEASURE 



EXAMPLE 



DRUG-FREE 
SCHOOLS (DFS) 
PROGRAM 


COST 


NUMBER OF 
STUDENTS* 
SERVED 


COST PER 
STUDENT* 


NUMBER OF STUDENTS 
PREVENTED FROM 
ALCOHOL AND OTHER 
DRUG (AOD) USE 
(EFFECT) 


COST PER 
STUDENT 
PREVENTED 
FROM AOD USE 
{COST /EFFECT) 


RATING 


Drug Abuse 
Resistance 
I Education (DARE) 


$43,298 


11,190 


$3.87 


1,119 


$39 


+ 



* Participants 



Cost: $43 f 298 

Number of Students Served: 1 1 , 1 90 

Cost Per Student: $3.87 ($43,298/11,190 - $3.87) 

Number of Students Prevented from Alcohol and Other Drug (A OD) Use (Effect): 1,119 
Students prevented from AOD use by the District Drug Free School (DFS) programs is 
calculated by subtracting the rate of use for the DFS program students (the recent use 
rate for DARE participants was 30%), from the average rate of use for all students in the 
District (40%). That difference (10%) was multiplied by the total number of students 
served by the program (11,190 * .10 = 1,119) 

Number of Students Prevented from 

Alcohol and other Drug (AOD) Use (Effect): 1,119 

Cost Per Student Prevented 

from AOD Use (Cost/Effect): $39 ($43,298/1,119 = $38.69 = $39 rounded) 
Rating: + 



Riling is expressed at contributing to any of the five AISD 
strategic objectivea. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needa to be improved and modified 
Negative, needa major modification or replacement 
Blank Unknown 



Coat is the expense over the regular District per atudcnl 
expenditure of about $4,000. 

0 No cost or minimal coat 

$ Indirect costs and overhead, but no aeparate budget 

$$ Some direct costs, but under S500 per student 

$$$ Major direct costs for Uachers. staff, and/or 

equipment in the range of $500 per student 
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Cost-Effectiveness of 1 992-93 Programs Based on a Drug Prevention Measure 



DRUG -FREE 
SCHOOLS IDFS) 
PROGRAM 


COST 


NUMBER OF 
STUDENTS* 
SERVED 


COST PER 
STUDENT* 


NUMBER OF STUDENTS 
PREVENTED FROM ALCOHOL 
AND OTHER DRUG (AOD) USE 
(EFFECT) 


COST Pen STUDENT 
PREVENTED FROM AOD 
USE (COST/EFFECT) 


RATING 


All Well Health 
Services 


$3,000 


10 staff 


$300 staff 


Insufficient information 






Conflict Resolution 
Project 


$33,352 


39 students 

57 staff 


$368 per 
participant 


0 






Drug Abuse Resistance 
Education (DARE) 


$43,298 


11,190 


$3.87 


1,119 


$39 


+ 


Innovative Programs 


$37,014 


932 


$39.71 


37 


$1,000 




K-1 2 Curriculum 


$47,186 


64,171 


$ .74 


Rating based on program 
records of service 




+ 


Medicine Education 
end Safety Program 


$5,772 






Rating based on completion of 
project and on reactions to 
conference presentations 




+ 


MagaSkills 


$21,798 
DPS 

$17,664 

Ch.z; 
$20,705 
Ch.1 


1 ,643 

parenta 

643 

students 


$13.27 per 
parent DFS 
$93.57 per 
student 
DFS 

$36.62 
(Total for 
all 

programs) 


109 


$200 


+ 


Peer Assistance and 
Leadership (PAL) 


9 DO, / 1 O 


1 044 


$52.81 


42 


$942 


+ 


Plays for Living 


$6,000 


4,472 


$1.34 


447 


$13 




Private Schools 


$18,143 


2,779 


$6.53 


Evaluation did not take place 
for this component. 






Quality Schools 


$160,45 


306 staff 

602 

students 


$524.35 
per staff 
trained 
$266.53 
per student 


42 


$3,820 


+ 


Student Alcohol and 
Drug Abuse Education 
and Prevention 
Program (SADAEPP) 


$94,433 
DFS 


2,488 


$37.96 


75 


$1,259 


+ 


$20,579 
Ch.2 


Student Assistance 
Program (SAP) 


$24,851 


185 staff 


$134.33 


Staff training was not evaluated 
this yeer. 







• Participants 
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Cost-Effectiveness of 1992-93 Programs Based on a Drop out Prevention Measure 

Another refinement in the methodology stems from the continuing review of the reporting of cost- 
effectiveness for dropout prevention programs. Since 1992-93, questions have persisted about the 
relationship among the number of students served, the number of at-risk students, the number of 
predicted dropouts, and the cost-effectiveness of dropout prevention programs. In the analyses on 
1991-92 dropout prevention programs, an exception to the method for calculating cost-effectiveness 
was made for those dropout prevention programs for which it could be argued that all the students 
were potential dropouts, regardless of their at-risk status based on academic risk factors. For these 
programs, the predicted dropout rate was set to 100%, rather than the percentage derived from the 
formula, and the obtained number of dropouts was computed on the total number of students served 
instead of the number of students whom the formula predicted would drop out. 

Similar modifications were made in 1 993-94 for 1 992-93 dropout prevention programs. To clarify the 
relationship among the number of students served, the number of at-risk students, and the number of 
predicted dropouts, the number of students who were at risk among the number of students served 
was added to the reporting. Likewise, allocation was prorated according to the numbers of at-risk 
students served and the total number of students served. Finally, cost-effectiveness was computed 
by dividing the cost of serving the at-risk students by the number of students prevented from dropping 
out. 

Definitions : Cost/Effect = Cost of the program for at-risk students/dropout prevention effect 

An example of the computation for a program using a dropout prevention effect measure, and 
reflecting the revised reporting format, follows on the next page. 
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COST-EFFECTIVENESS OF 1992-93 PROGRAMS BASED ON A DROPOUT 

PREVENTION MEASURE 



EXAMPLE 



PROGRAM 


ALLOCATION 
(COST) 


NUMBER OF 

STUDENTS 

SERVED 


COST 
PER 

STUDENT 


NUMBER OF 
DROPOUTS 

Predicted Obtained 


PREDICTED 
DROPOUTS 
WHO STAYED 
(N SCHOOL 
(EFFECT! 

# % 


COST PER 

STUDENT 

KEPT IN 

SCHOOL 

(COST/ 

EFFECT) 


RATING 


Newcomers Program 
Funding Source: External 
Grades: 9-12 


$126,000 


At risk: 1 34 
Total: 134 


$940 


134 6 


128 96 


$984 


+ 



Newcomers Program, 1 992-93 - Grades: 9-1 2 - Level of Service: 3 hours/day 
Cost: $126,000 

Number of Students Served: 1 34 

Cost Per Student: $940 [$126,000/134 = $940.30 = $940 rounded] 
Effect: 128 

[Predicted 134 students, Obtained 6 students] 
134- 6 = 128 students prevented from dropping out 
Cost/Effect: $984 [$126,000/128 = $984.38 = $984 rounded] 
Rating: + 

What this means is that it costs $984 for each student prevented from dropping out by the Newcomers 
Program who would otherwise have been expected to drop out of school. 
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METHOD 



ORE has conducted and reported cost analyses for a number of years, and this report builds on 
previous work (see -References-). Cost-effectiveness analysis was, however, a new venture, and ORE 
staff engaged in considerable discussion, over a period of months, about how cost-effectiveness 
should be calculated and how cost-effectiveness information should be integrated into ORE's annual 
report to AISD's Board of Trustees about program effectiveness. A first-person account of how staff 
thinking evolved and what decisions were made is detailed in "Notes on Cost Effectiveness," ORE 
Publication Letter 92. D. The following is a brief exposition of the method used in performing cost- 
effectiveness analyses on 1991-92 AISD programs. See "Definitions" and "Notes" for additional 
information. 

Following Levin (1983), cost-effectiveness is defined as cost divided bv effect: 
Cost/Effect (C/E) 

Cost was defined, per earlier ORE research (see Wilkinson, 1985), as a program's appropriation (i.e., 
budget). Cost was taken to include all funding for a program, regardless of source. Effect was defined 
either as (1 ) achievement or (2) not dropping out. 

Definitions : Cost = appropriation (budget) 

Effect = achievement, OR 

not dropping out 

The achievement measure of effect was operationalized as the residua/ (i.e., difference) between the 
achievement of the program students and some standard or expectation for their achievement. A 
standard against which to compare is necessary to distinguish between the effect of the special 
program and the effect pf the students' regular instructional program. Residual was defined as the 
difference between predicted and obtained scores, expressed in grade equivalents (GE's), from either 
the Norm-referenced Assessment Program for Texas (NAPT) or the Iowa Tests of Basic Skills (ITBS), 
both norm-referenced, standardized achievement test batteries. Three different residuals were 
identified: (1 ) average ROSE residual, (2) national norm gain residual, and (3) AISD gain residual. 

Definitions : Achievement = Average ROSE residual, OR 

National norm gain residual, OR 
AISD gain residual 

Residual = The difference between predicted and obtained score; for 
NAPT/ITBS, expressed in grade equivalents (GE's) 

Average ROSE residual = The average of the residuals from ROSE, on the 
reading and mathematics tests or the reading test alone, across grade levels, 
expressed in grade equivalents (GE's) 

National norm gain residua/ = The difference between observed gain and an 
expected gain of 1.0 GE per year on the average 

AISD gain residual = The difference between observed gain and the average gain 
in the District, in GE's 



32 38 



ROSE, the Report fin School Effectiveness, is a series of regression analyses that answer the question, 
"How do the achievement gains of a school's students compare with those of other AISD students 
of the same previous achievement levels and background characteristics?" ROSE predicts achievement 
scores for the group of students who have both pre- and posttest scores on the ITBS or the NAPT, 
depending on grade level and year of administration. Predictions are based on: 

Previous achievement level 
Sex 

Ethnicity 
Age 

Low-income status 
Family income 

Desegregation status of the school attended 
Whether or not the student was a transfer student 
Pupil-teacher ratio for school and grade 

The predicted scores are then compared with the students' actual scores. The difference between the 
predicted and actual scores is called the ROSE residual score, which is based on a GE score scale. If 
students' ROSE residual scores are far enough above or below zero to achieve statistical significance, 
they are said to have either "exceeded predicted gain M or to be "below predicted gain." Nonsignificant 
residual scores are classified as "achieved predicted gain." For more information about ROSE, see 
Paredes (1991). 

ORE'S GENeric Evaluation SYStem (GENESYS) produces, among other things, a Report on Program 
Effectiveness (ROPE). ROSE and ROPE are very similar, the major difference between them being that 
ROSE evaluates schools and ROPE evaluates programs. Most of the GE's used in calculations of 
achievement effect were obtained from ROPE analyses produced by GENESYS. GENESYS also 
produces, for each program run, counts of the number of students predicted to drop out and the 
number who dropped out (see below). For more information about GENESYS, see Ligon and Baenen 
(1989) and Wilkinson and Spano (1990). 

The dropout prevention measure of effect was operationalized as the difference between the number 
of students in a program predicted to drop out and the actual number of students who dropped out. 

Definition : Not dropping out = The difference between the number of students predicted 

to drop out, based on their at-risk category, and the actual 
number of dropouts 

Cost-effectiveness was operationalized as (1) cost per student divided by achievement effect, 
expressed in GE's or (2) cost of the program divided by dropout prevention effect (predicted minus 
actual dropouts). 

Definitions : Cost/Effect = Cost per student/achievement effect, OR 

Cost for the program/dropout prevention effect 

The cost-effectiveness ratio, expressed in dollars, which results from this division is a measure of the 
cost-effectiveness of a program, i.e., the amount of effect for monies expended, and because a 
common effect measure was used as the denominator among like programs, programs' cost- 
effectiveness can be compared. 

Definition : Cost/Effect = Cost-effectiveness ratio (in dollars) 
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Effect ratings were provided for programs (1) for which cost-effectiveness ratios could be calculated 
and (2) for which cost-effectiveness could not be calculated but about which other evaluation 
information was available. The ratings were based on the same scale which ORE had twice used 
previously. 

Pefinitions : Ratings: Same scale as in February 1992 program effectiveness charts; 

same as in ORE'S 1891-92 final reports: 

Effect is expressed as contributing to any of AlSD's five strategic 
objectives: 

+ Positive, needs to be maintained or expanded 

0 Not significant, needs to be improved and modified 

Negative, needs major modification or replacement 
blank Unknown, may have positive or negative impact on other 

indicators; however, impact on the five AISD strategic 

objectives is unknown. 

Cost is the expense over the regular District per-student 
expenditure. 
0 No cost or minimal cost 

$ indirect costs and overhead, but no separate budget 
$$ Some direct costs, but under $500 per student 
$$$ Major direct costs for teachers, staff, and/or equipment in 
the range of $500 per student or more 



Definitions : Effect Rating = + = Positive achievement gain, OR 

Number of students who actually dropped out was 
less than the number who were predicted to drop 
out, OR 

Positive opinion, based on other indicators, such as 
survey results, lower retention, or other success 

0 = Achievement gain less than 1 month, OR 
Neutral opinion 



Negative opinion, OR 

Number of students who actually dropped out 
exceeded the number who were predicted to drop 
out 



blank Insufficient information 
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DEFINITIONS 



At risk - In AISD, a student in grades 7-1 2 is considered at risk of dropping out if the student falls into 
one of 22 risk categories. 

Cpst - The total cost of the program, regardless of funding source. The cost of a program is above 
and beyond the cost of the regular educational program. In reporting costs, ORE standardly uses 
appropriation ox budget, not expenditure. Some programs have capital outlay costs, e.g., for computer 
equipment in a lab. These costs are shown as "investment cost," i.e., the initial cost of equipment 
and other items to get the program going. "Operating cost* 1 is the annual cost to keep the program 
functioning after large initial outlays have been made. Cost figures are rounded to the nearest dollar. 

Cost/effect - "Cost per student" or "cost" (for dropout prevention programs) divided by "effect." 
"Cost/effect" is the annual cost for one month's extra achievement gain above that attributable to the 
regular instructional program. 

Co$t-effpctiveness (C/E) analysis - A type of cost analysis concerned with the evaluation of 
alternatives according to both their costs and their effects with regard to producing some outcome or 
set of outcomes. In C/E analysis, a measure of cost is divided by a measure of effectiveness. This 
analysis is distinguished from other cost-effectiveness analyses by the measure used as the 
denominator. In cost-benefit (C/B) analysis, by comparison, the denominator is benefit expressed in 
dollars. 

Qopt; per student - "Cost" divided by "number of students served/ Service may have been provided 
to others besides students, e.g., teachers trained with Title II monies. In these instances, cost per 
participant should be understood. "Cost per student" is the numerator in the cost/effect calculation. 

Dropout - A student is reported as a dropout for a school year if the individual is absent for a period 
of 30 or more consecutive school days without approved excuse or documented transfer, or fails to 
reenroll by September 15 of the following school year without completion of a high school program. 

See "predicted dropout rate" and "obtained dropout rate." 

Drooout risk probability - Based on the risk factor associated with the student's membership in one 
of 22 different risk categories. 

See "risk category" and "risk factor." 

The probability that a student will drop out is based on the actual percentage of students in that risk 
category who have dropped out in the past. For example, if 42.66% of the students in risk category 
#12 dropped out the previous year, current-year students in that risk category would be assigned a 
dropout risk probability of 42.66. 

Effect - There are two measures of "effect." One is an achievement measure based on standardized 
test scores, and the second is a dropout prevention measure. All programs ultimately need to be held 
to the student achievement outcome criterion, even dropout and drug prevention programs. Like cost, 
the effect of a program, if any, is above that of the regular instructional program. 

The ROSE residurl (difference between predicted and obtained score) is the measure of achievement 
effect, unless the participants make up a disproportionate percentage of the comparison group. If the 
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program participants do make up a disproportionate part of the comparison group, another standard 
for comparison was selected. 

Options other than ROSE residuals include: 

• Actual gain expressed in grade equivalents, 

• National norm gain residual, the difference between observed gain and an expected gain 
of 1 .0 GE per year on the average, and 

• AISD gain residual, the difference between observed gain and the average gain in the 
District. 

For a program like DARE, for example, where all the 5th- and 7th-grade students are in the program, 
the only comparison available is the national norm. 

"Disproportionate" is defined as the program students making up 25% or more of the AISD students 
at that grade or achievement level. 

Achievement effect is expressed as a number greater than one (1). A GE gain of three months, for 
example, is expressed as 3.0, instead of 0.3. 

The ROSE (residual) or dropout measure (predicted minus obtained rate) is used as the effect for those 
programs for which these measures can be obtained. For other programs, a + /-/0/blank rating is 
assigned on the same basis as in past years' ORE reports. 

In the absence of a ROSE residual for the Composite test, the mathematics and reading residuals are 
averaged. 

The dropout effect is the "number of predicted dropouts who stayed in school," i.e., the number who 
did not c p out who were predicted to drop out. 

Funding source - Local, external, or both. External funding may be grant or other monies from other 
governmental entities or private organizations. 

Grades - The grade levels served by the program. Analyses are based on the grade levels for which 
measures are available. For example, although a program may serve grades K-6, districtwide 
achievement test scores are not available for kindergarten. 

Level of service - Generally reported in one of three categories--* D hours per week, 
(2) hours per day, or (3) full year-but may be more descriptive than quantitative. 

Number of, students served - May be enrollment in the program or the definition used in the evaluation 
last year. Not all programs serve students. In these instances, "number served" refers to participants. 

Obtained dropout rate - For a program or group, the actual percentage of students who dropped out. 

Predicted dropout rate - For a program or group, the sum of the dropout risk probability for each 
student in the group divided by the number of students in the group (N). The number of students 
predicted to drop out is not equivalent to the number of at-risk students. 

See "dropout risk probability," "risk category," and "risk factor." 
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For example, if the total of the students' risk factors for 90 students served by a dropout prevention 
program were 3,333.80, the predicted dropout rate would be 37.042, or 37.0% (3,333.80/90 = 
37.042). The predicted number of dropouts for the program would be 33 students (3,333.80/100 = 
33.338 = 33). In other words, of 90 students served, 33 (37.0%) would be predicted to drop out 
based on their dropout risk probabilities. 

The number of students predicted to drop out is not equivalent to the number of at-risk students 
because not all students who are at risk drop out, nor are all the students who drop out identified as 
at risk. 

Predicted number of dropouts - For a program or group, the sum of the dropout risk probability for each 
student in the group divided by 100. 

See "predicted dropout rate." 

Program - Includes any special activity customarily thought of as a program. Some programs, e.g., 
Chapter 2, have multiple program components. Programs often have separate budgets. 

Rating - A rating is supplied both for programs for which cost-effectiveness information can be 
provided and for programs about which ORE staff have an informed opinion based on evaluation 
information. In the former case, all programs which have a positive effect-defined as 0.1 GE (1 
month's gain in grade equivalents) or better-will have a + rating. (Because the cost-effectiveness 
ratio grows enormous the closer to zero effect size gets, it is impractical to report sizes smaller than 
0.1 GE). In the case of programs for which ORE does not have cost-effectiveness information but 
does have sufficient evaluation information for an informed opinion, the rating scale used in the 
program effectiveness summary pages in last year's ORE final reports is applied: 

Effect is expressed as contributing to any of the five AISD strategic objectives. 

+ Positive, needs to be maintained or expanded 
0 Not significant, needs to be improved and modified 
Negative, needs major modification or replacement 
Blank Unknown, may have positive or negative impact on other indicators; however, impact 
on the five AISD strategic objectives is unknown. 

Risk category - One of 22 used to identify and track at-risk secondary (grades 7-12) students. ORE 
extended the four state-mandated criteria to pinpoint differential dropout rates. Greater percentages 
of students in some risk categories drop out than in other risk categories. Additional, optional criteria 
for identifying at-risk students have been specified by the State, e.g., sexual, physical, or psychological 
abuse, living in a residential treatment facility, and being homeless. However, AISD does not maintain 
centralized files on students with these characteristics. Therefore, ORE does not use these criteria to 
identify at-risk students. 

Definitions of the secondary risk categories are attached. 

Risk factor - For a given risk category, the percentage of students in that risk category who dropped 
out. Expressed as a rate, the risk factor is a two decimal-place numeral. For example, if 45.75% of 
the students in a particular risk category dropped out, the risk factor for a student in that category 
would be 45.75. In other words, a student in this risk category would have almost a 50-50 chance 
of dropping out. 
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Category Factors 



Definitions of Secondary Risk Category Codes 

Definition 



Age 

Read Ach 



3 Math Ach 



Student is two or more years older than expected for the grade level 

Student scored two or more years below grade level in reading on a norm -referenced, standardized achievement 
test (either the Iowa Tests of Basic Skills or the Tests of Achievement and Proficiency) 

Student scored two or more year s below grade level in mathematics on a norm-referenced, standardized 
achievement test (either the ITBS or the TAP) 



4 
5 

6 
7 
8 
9 

10 

11 

12 

13 

14 

15 
16 

17 

in 

18 
19 

20 
21 



2Fs 

TEAMS Read 

TEAMS Math 

TEAMS Lang 

TEAMS Write 

TEAMS WComp 

Age, Read Ach or 
Math Ach 

Age,2Fs 

Age, TEAMS (any) 



Math Ach or 
Read Ach & 2 Fs 



Student failed at least two courses during a semester 



Student failed the reading section on the most recent administration of the Suite-mandated, criterion-referenced 
Texas Educational Assessment of Minimum Skills (TEAMS) (gradef 7 & 9 only) 

Student failed the mathematics section of the TEAMS 

Student failed the language arts section of the Exit-Level TEAMS (grades 1 1 &1 2 only) 
Student failed the writing section of the TEAMS (Grades 7 & 9 only) 

Student failed only the writing composition portion of the TEAMS Writing test (grades 7 & 9 only) 

Student is two or more years older than expected for the grade level and scored two or more years below grade 
level in reading or mathematics on the ITBS or TAP 

Student is two or more years older than expected for the grade level and failed at least two courses during a 
semester 

Student is two or more years older than expected for the grade level and failed at least one of the sections of the 
TEAMS 

Student scored two or more years below grade level in mathematics or reading on the ITBS or the TAP and failed 
at least two courses during a semester 



Math Ach or Read Student scored two or more years below grade level in mathematics or reading on the ITBS or the TAP and failed 

Ach & TEAMS (any) at least one of the sections of the TEAMS 

2 Fs, TEAMS (any) Student failed at least two courses during a semester and failed at least one of the sections of the TEAMS 

Age, Math Ach or Read Student is two or more years older than expected for the grade level, scored two or more years below grade level 

Ach, & 2 Fs in mathematics or reading on the ITBS or the TAP, and failed at least two courses during a semester 

Age, Math Ach or Read Student is two or more years older than expected for the grade level, scored two or more years below grade level 

Ach, & TEAMS (any) in mathematics or reading on the ITBS or the TAP, and failed at least one of the sections of the TEAMS 



Age, 2 Fs, & 
TEAMS (any) 

Age, Math Ach or 
Read Ach,2Fs, 
& TEAMS (any) 

Math Ach & 
Read Ach 

TEAMS (two) 



Student is two or more years older than expected for the grade level, failed at least one of the sections of the 
TEAMS 

Student is two or more years older than expected for tlie grade level, scored two or more years below grade level 
in mathematics or reading on the ITBS or the TAP, failed at least two courses during a semester, and failed at 
least one of the sections of the TEAMS 

Student scored two or more years below grade level in mathematics and in reading on the ITBS or the TAP 
Student failed at least two sections of the TEAMS 



22 Math Ach or 
Read Ach, 2 Fs, 
& TEAMS (any) 



Student scored two or more years below grade level in mathematics or reading on the ITBS or the TAP, failed at 
least two courses during a semester, and failed at least one of the sections of the TEAMS 



Note: "TEAMS" should be interpreted as "TEAMS/TAAS." 
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NOTES 



Page 2 

1. See Levin, H. M. (1983). Cost-effectiveness: A primer. Beverly Hills, CA: Sage Publications. 
Page 8 

1 . The zero (0) rating for Bilingual/ESL programs derives from the nonsignificant achievement effect (less than 
an average of 1 .0 grade equivalents for reading and mathematics) for served students (which does not include 
students who qualified for services but whose parents denied services). Interpretation of this effect, however, should 
take into account the relatively few limited-English-proficient (LEP) students for whom there were test scores. At 
grades pre-K through 6, approximately 1 6% of the LEP students served had both pre- and posttest scores. At grades 
6-8, 32% of the students served had test scores, and at grades 9-12, 40% of the students served had test scores. 

Page S 

1. Funding for the Elementary Computer Lab at Blanton came from Chapter 2 ($16,522) and, according to the 
administrative supervisor of Instructional Technology, the local budget contributed $40,000 for software. 

2. Report on Program Effectiveness (ROPE) results were used as an effect measure for the Bridge computer lab 
at Read and for the Wicat computer lab at Blanton. Calculations were made using the procedures outlined in Notes 
on Cost Effectiveness Ul (see ORE Publication Letter 92.D). 

3. The amount shown for estimated investment cost for the Kealing Magnet Program derives from budget 
summary data sheets from 1985-86 provided by Finance in March 1993. According to these sheets, the Kealing 
Magnet Program was allocated £10,000 in 1986-87 for purchased services, capital outlay, and supplies; Kealing 
Junior High School opened during the second semester of 1 986-87. The sheets also indicate that the 1 985-86 budget 
provided planning time for the principal of Kealing for the year before the school opened. Presumably, some salary 
costs for the principal for that year could also be included in the investment cost for the Kealing Magnet Program. 

Page 10 

1 . The amount shown for estimated investment cost at the Liberal Arts Academy (LAA) was supplied by LAA staff 
in March 1993; it is the allocation for 1987-88, the year before the LAA opened, which was a planning year. That 
year, funds were allocated for a magnet planner, a curriculum coordinator, a secretary, teacher stipends, instructional 
supplies, travel, and capital outlay (see AISD budget book for 1987-88). 

Pages 10-11 

1. Costs for the Project A + Elementary Technology Schools break down as follows: 
Annual Operating Costs 





Patton 


Andrews 


Langford 


Galindo 


Lab techs 


$28,527 (3) 


$28,527 (3) 


$19,018 (2) 


$ 9,509 (1) 


Other 


$34,726 


$34,726 


$34,726 


$34,726 


Total 


$63,253 


$63,253 


$53,744 


$44,235 



( ) = Number of people 
Investment Cost 

IBM: $3,207,300 



Patton 

Andrews 

Langford 



1,354,320 
1,100,956 
749,642 



($138,905/4 « $34,726.25) 



weighted by size of school 
for three IBM schools 
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Apple: 10,000 software 

74,000 equipment 

1 1 2,000 AISD contribution 

50.000 cablinQ 

Galindo $ 246,000 

Galindo's fiQures reflect one-time only costs using used equipment and could not be duplicated at another school. 
Page 1 1 

1 . The investment cost for the Science Academy of Austin was determined as follows. Students first began 
attending the Science Academy in 1985-86; the previous year was developmental. According to AISD's 1985-86 
budget book, the 1984-85 budget for the Science Academy was $270,900, which provided salaries, purchased 
services, supplies, other operating costs, and $40,000 in capital outlay. This amount was added to a $242,811 
Department of Education grant in 1985-86 (figure provided by Science Academy staff) for a total of $513,71 1. 

Page 1 3 

1 . As an alternative school, the Alternative Learning Center (ALC), the whole school, has long been thought of 
as a dropout prevention program. Costs for the ALC break down as follows: 

Fund 112 Local $1,054,527 
Fund 322 Federal Vocational 1 ,400 

Fund 382 Chapter 2 1,944 

$1,057,871 

Both local and external funds were included. Capital improvement costs were not included. 

Costs were obtained from a budget status printout (FINB21S) run 1/6/93 for the period ending 8/31/92 supplied by 
Internal Audit. 

Because separate dropout rates are calculated for grades 7-8 and grades 9-1 2, and because the ALC serves students 
across those grade levels, costs were prorated across the two grade spans based on the numbers of students; thus, 
$429,760 represents 40.625% of the cost of the ALC (104 students in grades 7-8 divided by 256 students 
altogether), and $628,1 1 1 is the remaining 59.375% (1 52/256). By apportioning costs by grade span, the cost per 
student, $4,132, is the same for grades 7-8 as for grades 9-12. 

2. As regards the ALC and predicted dropout rates, two major points must be kept in nrvnd. First, when we 
predict the dropout rate of a group of students who are selected into the program specifically because they are at risk 
of dropping out, and when we use the difference between predicted rate and actual rate as a measure of program 
effect, we are confounding the imprecision of our dropout prediction with program effect. In other words, some of 
the differences we see may be due to imprecision in dropout prediction rather than differences among programs. This 
alternative explanation is true of any such analysis, but is magnified in this case. The imprecision is more of a concern 
here because the prediction is derived from the student population as a whole and then applied to a restricted, 
nonrandom sample. For this reason, where programs select at-risk students only, the predicted dropout rate is set 
to 100% since all students in the program should be at risk of dropping out whether our formula predicts it or not. 
Second, as our dropout prevention programs do a better job of keeping students in school, there may be a weakening 
of the relationship between the predictors we use and the probability of dropping out. We must continue to examine 
any dropout prediction formula to see how well it is performing. As our ability to predict decreases, more of the 
differences we see among programs will be due to random or unmeasured effects rather than program effectiveness. 
However, this likely future decrease does not negate the present usefulness of comparing actual numbers of dropouts 
with some predicted number in measuring program effectiveness. 

3. No allocations are shown for the Block- Programs because these programs involved a reorganization of local 
campus resources, not additional funding. Effectively, the cost for these programs was zero. 

4 The positive ( + ) ratings for Block Programs are based on the programs having kept in school students who 
were predicted to drop out. Because costs could not be obtained, cost-effectiveness could not be calculated; 
however, effect ratings could still be made. 
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Page 14 

1. All of the students in the Communities in Schools (CIS) program at Robbins and at the Evening School are at 
risk by definition. See Note 2 to page 1 3. 

2. The Evening School as a whole is thought of as a dropout prevention program. Costs for the Evening School 
break down as follows: 

Fund 112 Local $328,846 
Fund 322 Federal Vocational 500 



$329,346 

Both local and external funds were included. Capital improvement costs were not included. 

Costs were obtained from a budget status printout (F1NB21S) run 1/6/93 for the period ending 8/31/92 supplied by 
Internal Audit. 

2. The Johnston Technology Learning Center (TLC) is a Chapter 2-funded program. See page 22 for other 
information about the program. 

Page 15 

1 . See page 25 for more information about the Title VII secondary bilingual program called the Newcomers 
Program. 

Page 16 

1 . As an alternative school, Robbins Secondary School, the whole school, has long been thought of as a dropout 
prevention program. Costs for Robbins break down as follows: 



Fund 112 Local $1,130,696 

Fund 322 Federal Vocational 76,420 

Fund 382 Chapter 2 8,944 

Fund 472 Teen Parent 1 1 7,1 78 



$1,333,238 

Both local and external fur>u? were included. Capital improvement costs were not included. 

Costs were obtained from a budget status printout (FINB21S) run 1/6/93 for the period ending 8/31/92 supplied by 
Internal Audit. 

2. Costs for Zenith were obtained from a budget printout as of 8/31/92. Costs under subobject .7F were totaled 
across organizations. Transactions for organizations 016 and 268, both codes for Evening School, were totaled: 
$122,790 + $9,200 = $131,990. No other Zenith costs appeared with this search strategy. 

Pages 18-20 

1. Except for the prekindergarten program, ratings for all Chapter 2 programs were taken from the Chapter 2 
Formula 1991-92 final report. 

2. For the prekindergarten program, the rating was based on previous years' test results, since the validity of the 
Bracken Basic Concepts Scale (BBCS) test results in 1991-92 was questionable. 

3. For all Chapter 2 programs, "number of students served" was taken from Taken from the Chapter 2 Formula 
1991-92 final report. 

4. The cost of Chapter 2 programs was based on actual allocations taken from December 16, 1991 Chapter 2 
Formula Budget Amendment #1. 

5. For computer labs at Blackshear, Blanton, Read, and Johnston, investment costs were obtained from the 
administrative supervisor for Instructional Technology, and are estimates. 
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6. Because there were no achievement data for these Chapter 2 programs, ratings were based on the following 
indicators: 



Academic Decathlon 

Writing to Read computer lab at Blackshear 
Extracurricular Transportation 
Library Resources 

Multicultural/Special Purpose Buses 
Private Schools 

Secondary Library Technology Support 
Spanish Academy 
Support for Restructured Robbins 
Technology for Access to Problem Solving 
Technology Learning Center at Johnston 



Employee survey 

Employee survey 

Employee and student survey 

Employee survey 

Employee and bus user survey 

Private School survey 

Purchases 

Participant survey 

Principal interview 

Employee survey 

Employee survey 



Page 29 

1 . Some of the Drug-Free Schools (DFS) programs included participants who may have been more at risk for 
alcohol and other drug (AOD) use than the average District student. Therefore, some of the estimates of students 
prevented from AOD use may be conservative. 

2. Note that only DFS cost was used in cost-effectiveness calculations. Some programs received additional 
funding from other sources. 

3. Percentages for recent AOD use by program participants and their differences from the District average were 
as follows: 

Percentage Points Better 
Recent AOD Use by Than the District 
DFS Program Program Participants Average Rate of Use (40%) 

Drug Abuse Resistance Education (DARE) 
Innovative Programs 
MegaSkills 

Peer Assistance and Leadership (PAL) 
Plays for Living 
Quality Schools 

Student Alcohol and Drug Abuse Education 
and Prevention Program (SADAEPP) 

Page 34 

1 . AISD's five strategic objectives, published in Austin independent School District Strategic Plan 1992-1997, 

are: 

■ Every student will function at his/her optimal level of achievement and will progress successfully through the 
system. 

■ All students will function successfully at or above international standards. 

■ One hundred percent of all students who enter AISD will graduate. 

■ After exiting AISD, all individuals will be able to perform successfully at their next endeavor. 

■ AISD will upgrade the qualiaty of course content and the effectiveness of instruction. 



30% 
36% 
23% 
36% 
30% 
37% 
37% 



10 
4 
17 
4 

10 

3 
3 
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